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PERFORMANCE  ENHANCEMENT  WITH  REAL-TIME 
PHYSIOLOGICALLY  CONTROLLED  ADAPTIVE  AIDING 


Glenn  F.  Wilson*,  Jared  D.  Lambert^  and  Chris  A.  Russell* 

*Air  Force  Research  Laboratory,  WPAFB,  OH,  US 
^Sytronics,  Inc.,  Dayton,  OH,  US 

In  contemporary  systems  the  functional  state  of  the  operator  is  not  considered  during  system 
operation.  Degraded  states  of  operator  functioning  can  result  from  the  demands  of  controlling 
complex  systems,  the  work  environment  and  internal  operator  variables.  This,  in  turn,  can  lead 
to  errors  and  overall  suboptimal  system  performance.  In  the  case  of  mental  workload,  system 
performance  could  be  improved  by  reducing  task  demands  during  periods  of  operator  overload. 
Accurate  estimation  of  the  operator’s  functional  state  is  crucial  to  successful  unplementation  of 
an  adaptive  aiding  system.  One  method  of  determining  operator  functional  state  is  by  monitoring 
the  operator’s  physiology.  In  the  present  study,  physiological  signals  were  used  to  continuously 
monitor  subject’s  functional  state  and  to  adapt  the  task  by  reducing  the  number  of  subtasks  when 
high  levels  of  mental  workload  were  detected.  The  goal  was  to  demonstrate  performance 
improvement  with  adaptive  aiding.  Because  adaptive  aiding  during  high  mental  workload  has 
not  been  previously  implemented  its  benefit  has  not  be  demonstrated.  Application  of  adaptive 
aiding  techniques  reduced  tracking  task  error  by  44%  and  resource  monitoring  error  by  33%. 
These  results  demonstrate  the  utility  of  adaptive  aiding  using  physiological  measures  with 
artificial  neural  networks  to  determine  the  appropriate  time  to  introduce  the  aiding. 


INTRODUCTION 

Today’s  complex  systems  can  place  high  cognitive 
demands  upon  their  operators.  The  rate  of  inforrhation 
flow,  complex  nature  of  this  information,  and  the 
required  decisions  can  become  overwhelming  to  the 
human  operator.  On  the  other  hand,  automation  of  tasks 
can  lead  to  complacency  and  errors  of  inattention. 
Current  systems  are  capable  of  modifying  themselves  to 
meet  the  current  needs  of  the  operator.  This  includes 
taking  over  some  functions  for  the  operator  until  the 
mental  load  is  reduced.  In  other  cases  systems  can 
adjust  in  order  to  engage  the  operator  to  relieve  boredom 
or  inattention.  In  each  case  the  critical  need  is  the 
accurate  and  reliable  assessment  of  operator  state. 
Adaptive  aiding  by  the  system  can  be  beneficial  only 
when  supplied  at  the  appropriate  time  and  with  the 
consent  of  the  operator. 

One  method  of  determining  operator  functional 
state  is  by  monitoring  the  operator’s  physiology. 
Numerous  examples  of  physiological  measures 
providing  operator  state  assessment  are  available.  It  has 
been  demonstrated  that  the  various  physiological 
measures  provide  unique  information  about  several 
aspects  of  operator  state.  Eye  blink  rate  contains 
valuable  information  about  the  visual  demands  of  tasks. 
Heart  rate  is  of  use  to  determine  the  overall  engagement 
of  an  operator  by  a  task  (Wilson  &  Eggemeier,  1991). 


Because  of  the  nature  of  physiological  signals  they  are 
always  present  and  thereby  provide  continuous 
information  about  operator  state. 

Several  studies  have  used  physiological  measures  to 
classify  operator  state  related  to  mental  workload.  Most 
of  these  studies  employed  electroencephalo-graphic 
(EEG)  data  These  studies  used  either  simple,  single  task 
paradigms  (Gevins,  Smith,  Leong,  McEvoy,  Whitfield, 
Du  and  Rush,  1998;  Gevins  &  Smith,  1999;  Nikolaev, 
Ivanitskii  and  Ivanitskii,  1998;  Russell  &  Wilson,  1998; 
Wilson  &  Fisher,  1995)  or  used  relatively  few  peripheral 
physiological  variables  with  complex  task  performance 
(Wilson  &  Fisher,  1991).  They  report  overall  successful 
task  classification  in  the  high  80  to  90  percent  correct 
range.  These  excellent  results  classifying  high  mental 
workload  or  altered  operator  state  are  very  encouraging. 
They  suggest  that  these  methods  could  be  used  to 
provide  accurate  and  reliable  operator  state  assessment 
to  an  adaptive  aiding  system.  Psychophysiological 
measures  have  been  used  to  implement  adaptive  aiding 
in  laboratory  situations  of  lowered  operator  engagement 
(Freeman,  Mikulka,  Prinzel.  &  Scerbo,  1999;  Pope, 
Comstock,  Bartolome,  Bogart  and  Burdette,  1995; 
Prinzel,  Scerbo,  Freeman  and  Mikula,  1997).  These 
investigations  showed  enhanced  operator  performance 
when  the  adaptive  aiding  system  detected  subject 
disengagement  and  modified  the  task  to  increase  subject 
involvement. 


In  the  case  of  mental  overload,  system  performance 
could  be  improved  by  reducing  task  demands.  Accurate 
estimation  of  the  operator’s  functional  state  is  crucial  to 
successful  implementation  of  an  adaptive  aiding  system. 
In  the  present  investigation  physiological  signals  were 
continuously  monitored  in  order  to  determine  the 
subject’s  functional  state  and  to  adapt  the  task  when 
high  levels  of  mental  workload  were  detected.  Adaptive 
aiding  during  periods  of  high  mental  workload  has  not 
been  accomplished  before  using  physiological  measures. 
The  goal  of  the  present  study  was  to  determine  the 
effects  of  adaptive  aiding  upon  task  performance  when 
the  aiding  was  provided  during  periods  of  high  mental 
workload.  While  improved  scores  are  expected  it  is 
possible  that  the  aiding  may  interfere  with  subject 
performance. 

METHODS 

Seven  subjects  (4  female)  participated  in  the 
experiment.  Their  age  range  was  from  19  to  26  years. 
They  were  trained  to  stable  performance  on  the  NASA 
Multiple  Attribute  Task  Battery  (MATB,  Comstock  & 
Amegard,  1992).  After  initial  familiarization  with  the 
task  they  were  trained  to  use  their  right  hand  to  control  a 
joystick  and  their  left  hand  to  control  a  mouse.  All  of  the 
MATB  subtasks  were  used,  lights  and  dials  monitoring, 
tracking,  resource  management  and  the  auditory  task. 
Two  levels  of  difficulty  were  used  and  was  manipulated 
by  varying  the  number  of  events  that  occurred  during 
each  of  the  five  minute  trials.  Performance  scores  from 
each  task  were  recorded  and  practice  was  maintained 
until  each  subject  exhibited  stable  performance  on  all 
tasks.  This  required  approximately  six  hours  of  practice 
spread  over  three  days. 

Physiological  data  were  recorded  during  task 
performance.  The  physiological  data  consisted  of  six 
EEG  channels,  electrocardiographic  (ECG), 
electrooculographic  (EOG)  and  respiration  inputs.  On 
the  day  of  data  collection  the  subjects  practiced  the  tasks 
for  5  minutes.  Then  three  conditions  were  presented  to 
the  subjects.  The  first  was  a  resting  condition  during 
which  the  subjects  merely  looked  at  the  screen  .  The 
second  condition  required  them  to  perform  the  task  at 
the  low  workload  level.  During  the  third  condition  the 
task  was  presented  at  the  higher  difficulty  level.  Using 
the  physiological  data  from  these  three  conditions  an 
artificial  neural  network  (ANN)  was  trained  to  recognize 
these  three  conditions.  The  input  to  the  ANN  consisted 
of  spectral  EEG  features  including  delta  (1-3  Hz), 
theta  (4-7  Hz),  alpha  (8  -  13  Hz)  and  beta  (14  -  30  Hz) 
bands.  Other  features  included  ECG  interbeat  intervals, 


blink  rate  and  blink  closure  duration,  and  respiration 
rate.  One  half  of  the  data  from  each  of  the  three 
conditions  were  randomly  selected  and  used  as  training 
data.  The  remaining  half  was  used  as  test  data  to 
determine  the  accuracy  of  the  ANN. 

The  trained  ANN  was  used  to  determine,  on-line, 
which  of  the  three  conditions  the  subjects  was 
performing  based  entirely  on  the  physiological  data.  The 
three  conditions  used  for  training  were  repeated  twice. 
During  both  replications,  on-line  determination  of 
subject  workload  level  was  performed  each  second.  In 
the  first  case  the  workload  classifications  were  recorded 
to  determine  the  accuracy  of  the  trained  ANN.  In  the 
second  case  adaptive  automation  was  applied.  When  the 
high  workload  condition  was  detected  by  the  trained 
ANN  the  MATB  task  was  adapted  by  “turning  off’  two 
of  the  subtasks.  During  adaptive  aiding,  the  lights  and 
dials  monitoring  and  the  auditory  task  were  “turned  off’ 
and  their  areas  on  the  screen  were  highlighted  in  blue  to 
indicate  that  an  aiding  period  was  in  progress.  The 
subjects  were  instructed  to  ignore  these  tasks  and 
concentrate  their  efforts  on  the  tracking  and  resource 
management  tasks.  They  were  given  practice  ‘ignoring’ 
these  tasks. 

MATB  performance  scores  were  recorded  so  that 
the  effects  of  the  adaptive  aiding  could  be  evaluated. 

RESULTS 

The  accuracy  of  the  trained  ANN  was  first  tested  by 
having  it  classify  the  test  set  of  the  original  data  run. 

The  ANN  accuracy  for  the  training  data  was  98.5% 
correct.  This  almost  perfect  accuracy  of  classifying  the 
baseline,  low  and  high  workload  conditions  is  typical. 
During  the  first  test  run  where  the  ANN  was  used  to 
only  classified  the  workload,  very  high  levels  of 
classification  accuracy  were  also  achieved.  The 
accuracy’s  were  84.9%  for  the  baseline,  82.0%  for  the 
low  workload  and  86%  for  the  high  workload 
conditions.  See  Table  1.  These  results  demonstrate  that 
an  ANN  can  produce  high  levels  of  correct  classification 
while  subjects  perform  complex  multiple  task  scenarios. 
Noteworthy  from  the  table  is  the  pattern  of  confusion  by 
the  ANN  of  adjacent  workload  conditions  when 
misclassifications  did  occur.  During  the  baseline 
condition  the  vast  majority  of  errors  were  assigned  to 
the  low  condition.  The  errors  during  the  low  condition 
were  primarily  confusion  with  the  baseline  condition 
and  most  of  the  errors  for  the  high  condition  were  miss 
categorized  as  belonging  to  the  low  condition. 


Classification  accinracies  for  each  subject  showed  an 
overall  mean  correct  classification  range  from  69.0% 
to  97.8%.  See  Table  2.  The  highest  accuracy  for  any 
one  condition  was  100%.  All  of  the  observed  accuracies 
were  well  above  the  expected  chance  level  of  33%. 


was  performed  by  the  subjects.  The  mean  correct 
classification  across  subjects  ranged  from  82.0%  to 
86.0%.  These  results  are  consistent  with  previous 
reports  and  demonstrate  the  high  levels  of  accuracy  that 
are  possible  using  ANNs.  No  doubt  using  both  central 
and  peripheral  nervous  system  features  enhanced  the 
performance  of  the  ANN.  Additional  accuracy  may  be 
achieved  by  including  performance  features  when 
possible  (Wilson  &  Russell,  1999). 

The  utilization  of  operator  state  information  to 
govern  the  application  of  adaptive  aiding  also  provided 
remarkable  results.  Lowering  task  demands  based  upon 
operator  state  resulted  in  large  enhancements  in  operator 
performance.  The  tracking  task  error  was  reduced  by 
44%  and  the  resource  management  error  was  reduced  by 
33%.  Reduction  of  overall  task  demands  by  temporarily 
removing  the  burden  of  the  monitoring  and  auditory  task 


based  upon  the  physiologically  determined  operator 


Table  2.  Percentage  correct  classification  for  the  three  conditions  during  aiding  session 
for  each  of  the  eight  subjects. 


SI 

S2 

S3 

S4 

S5 

S6 

S7 

MEAN 

BASE 

100.0 

85.0 

85.0 

93.3 

100.0 

90.0 

41.0 

84.9 

LOW 

95.0 

86.7 

28.7 

76.7 

91.7 

100.0 

95.0 

82.0 

HIGH 

98.3 

81.7 

93.3 

41.3 

93.3 

100.0 

94.3 

86.0 

MEAN 

97.8 

84.4 

69.0 

70.4 

95.0 

96.7 

76.8 

84.3 

During  adaptive  aiding  the  subjects  performance  on 
the  tracking  and  resoiurce  management  tasks  was 
monitored.  Because  the  purpose  of  the  aiding  is  to 
permit  the  subjects  to  concentrate  on  these  two  tasks 
during  epochs  of  high  workload  one  would  expect 
improved  performance.  By  removing  the  monitoring 
and  audio  tasks  the  subjects  were  free  to  focus  their 
efforts  on  the  remaining  two  tasks.  This  was  exactly  the 
case.  Adaptive  aiding  resulted  in  a  44%  reduction  in 
RMS  tracking  error  (t  =  -6.134,  p  <  0.0008)  compared 
to  the  nonadaptive  condition.  Performance  on  the 
resource  management  task  improved  with  a  33% 
reduction  in  the  error  score  that  was  marginally 
significant  (t  =  -1.822,  p  <  0.06). 

DISCUSSION 

These  results  demonstrate  that  an  ANN  using 
central  and  peripheral  nervous  system  features  can  be 
trained  to  very  accurately  determine,  on-line,  the 
functional  state  of  an  operator.  This  is  especially 
significant  in  light  of  the  complex,  multiple  task  that 


state  freed  the  subjects  to  concentrate  on  the  two 
remaining  tasks  and  greatly  improve  their  performance. 

The  effect  of  removing  the  two  subtasks  without 
regard  to  the  subject’s  state  remains  to  be  determined. 
Randomly  removing  these  subtasks  could  also  have 
improved  performance.  However,  it  is  possible  that 
there  would  have  been  no  change  or  degraded 
performance  from  the  random  removal  of  the  two 
subtasks.  Random  removal  might  interfere  with  the 
subject’s  strategy  and  lead  to  deteriorated  performance. 
This  question  will  have  to  be  determined  by  further 
research. 

Because  the  physiological  features  are  continuously 
available  on-line,  real  time  functional  state  monitoring 
systems  can  be  developed.  Further  research  will  be 
needed  to  find  the  best  methods  for  determining  the 
parameters  of  the  ANN  and  to  enhanced  feature 
selection.  Other  issues  still  remaining  to  be  resolved 
before  ANNs  can  be  routinely  applied  in  the  work  place. 
These  issues  have  to  do  with  their  day  to  day  reliability 
and  the  necessity  to  establish  a  ANN  for  each  subject 


rather  that  using  a  generic  solution  that  would 
accommodate  all  operators. 
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